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Executive  Summary 


The  U.S.  Army  Strategic  Software  Improvement  Program  (ASSIP)  is  a  multiyear  effort  to 
improve  the  way  the  Army  acquires  software-intensive  systems.  As  part  of  the  ASSIP,  the 
Carnegie  Mellon  Software  Engineering  Institute  examined  12  of  the  Army’s  Acquisition 
Category  1  (ACAT  1)  programs  using  a  method  called  Benchmarking  for  Improvement 
(BFI).  The  purpose  of  conducting  the  BFI  engagements  was  to  define  the  current  state  of 
acquisition  practices  across  the  Army  to  discover  best  practices  currently  in  use  by  Army 
program  offices,  to  identify  the  software  challenges  that  extend  across  Army  programs,  and  to 
brainstorm  potential  solutions  and  recommendations  for  Army- wide  improvement.  In  addi¬ 
tion,  the  BFI  team  provided  each  program  manager  (PM)  with  an  independent  view  of  pro¬ 
gram-level  activities  and  made  specific  recommendations  for  improvement.  A  briefing  pro¬ 
vided  to  each  program  manager  documented  these  recommendations. 

This  report  documents  the  results  of  the  interviews  conducted  during  BFI  engagements.  The 
following  list  provides  a  high-level  view  of  some  of  the  themes  that  surface  during  the  inter¬ 
views: 

•  Risk  management  practices  are  not  standardized  and  risks  are  inconsistently  and  insuffi¬ 
ciently  tracked,  updated,  and  addressed. 

•  The  acquisition  policy  changes  that  occurred  during  the  past  five  years  have  created  con¬ 
fusion  and  difficulties  that  are  exacerbated  by  operational  demands  for  rapid  delivery  of 
early  capability. 

•  Oversight  and  monitoring  of  contractors’  system  engineering  and  management  practices 
is  not  executed  consistently. 

•  Program  management  offices  do  not  employ  personnel  who  have  the  specialized  skills 
needed  to  respond  to  all  of  the  demands  of  their  jobs.  In  addition,  TRADOC  (Training 
and  Doctrine  Command)  System  Manager  (TSM)  groups  are  understaffed. 

•  Program  management  offices  perceive  Office  of  the  Secretary  of  Defense  (OSD)  and  De¬ 
partment  of  the  Army  (DA)  policy  and  directives  as  being  in  constant  flux,  which  makes 
it  difficult  to  locate  or  develop  interpretation  expertise. 

•  Program  management  certification  does  not  sufficiently  recognize  the  value  of  develop¬ 
mental,  operational,  TSM,  DA,  or  OSD  assignments. 

•  The  use  of  Integrated  Product  Teams  (IPTs)  is  mandated;  however,  some  programs  do  not 
execute  them  effectively. 
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Abstract 


The  U.S.  Army  Strategic  Software  Improvement  Program  (ASSIP)  is  a  multiyear  effort  to 
improve  the  way  the  Army  acquires  software-intensive  systems.  As  part  of  the  ASSIP,  the 
Carnegie  Mellon  Software  Engineering  Institute  (SEI)  examined  12  of  the  Army’s  Acquisi¬ 
tion  Category  1  programs,  using  a  method  called  Benchmarking  for  Improvement  (BFI).  The 
purpose  of  conducting  the  BFI  engagements  was  to  define  the  current  state  of  the  acquisition 
practices  across  the  Army,  to  discover  best  practices  currently  in  use  by  Army  program  of¬ 
fices,  and  to  identify  the  software  challenges  that  extend  across  Army  programs.  As  part  of 
the  BFI  process,  the  program  management  office  and  SEI  interview  teams  worked  together  to 
identify  Department  of  Army  best  practices  and  shortcomings  in  the  overall  acquisition  proc¬ 
ess,  as  well  as  potential  solutions  and  recommendations.  In  addition,  the  SEI  team  provided 
each  program  manager  with  an  independent  view  of  program-level  activities  and  made  spe¬ 
cific  recommendations  for  improvement.  A  briefing  provided  to  each  program  manager 
documented  these  recommendations. 

This  report  documents  the  results  of  the  interviews  conducted  during  BFI  engagements. 

These  results  are  of  interest  to  Program  Executive  Office  staffs,  Program  Management  Office 
staffs,  and  Department  of  Army  staffs  that  are  involved  in  acquisition. 
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1  Introduction 


In  late  2002,  the  Carnegie  Mellon  Software  Engineering  Institute  (SEI)  entered  into  a  strate¬ 
gic  agreement  with  the  assistant  secretary  of  the  Army  for  acquisition,  logistics,  and  technol¬ 
ogy  (ASA(ALT)).  This  partnership,  known  as  the  Army  Strategic  Software  Improvement 
Program  (ASSIP),  seeks  to  improve  the  Army’s  techniques  for  acquiring  systems  with  high 
software  content,  called  software-intensive  systems, 1  or  SIS. 

This  special  report  presents  the  results  of  the  AS  SIP-sponsored  examination  of  12  of  the 
Army’s  Acquisition  Category  (ACAT)  1  programs.  Examinations  were  conducted  from  July 
2003  to  October  2004  using  a  method  called  Benchmarking  for  Improvement  (BFI).  The  BFI 
method  was  designed  to  define  the  current  state  of  the  acquisition  practices  across  the  Army, 
to  discover  best  practices  currently  in  use  by  Army  program  management  offices  (PMOs), 
and  to  identify  the  software  challenges  that  extend  across  Army  programs.  In  addition,  each 
Program  Manager  (PM)  was  provided  an  independent  view  of  program-level  activities  and 
specific  recommendations  for  improvement,  which  were  documented  in  a  briefing.  The  SEI 
is  using  this  information  to  help  guide  the  ASSIP  improvement  initiatives.  By  sharing  this 
information  with  the  acquisition  functions  of  the  other  services,  other  government  agencies, 
and  industry,  the  SEI  and  the  ASSIP  hope  to  stimulate  open  discussions  about  SIS  acquisition 
issues. 

In  order  to  understand  these  results,  a  brief  review  of  the  ASSIP  is  included  in  the  next  sec¬ 
tion.  In  addition,  Sections  1.1.1  and  1.1.2  summarize  the  results  of  other  ASSIP  data  collec¬ 
tion  efforts  to  help  the  reader  develop  a  broader  picture  of  the  impact  that  acquisition  im¬ 
provement  efforts  are  having  on  the  Department  of  the  Army  (DA). 

1 .1  The  Army  Strategic  Software  Improvement 
Program 

The  ASSIP  is  a  multiyear  effort  targeted  at  dramatically  improving  the  way  the  Army  ac¬ 
quires  software-intensive  systems.  The  ASA(ALT)  initiated  the  ASSIP  in  2002  as  a  means  of 
helping  the  Army  proactively  respond  to  the  challenges  of  developing  systems  that  are  in¬ 
creasingly  dependent  on  software.  Later,  when  Congress  included  Section  804  in  the  Na¬ 
tional  Defense  Authorization  Act  for  Fiscal  Year  2003,  which  required  the  services  to  develop 


1  According  to  the  Defense  Acquisition  University  (DAU),  a  software-intensive  system  is  one 
in  which  software  represents  the  largest  segment  in  one  or  more  of  the  following  criteria:  system 
development  cost,  system  development  risk,  system  functionality,  or  development  time  [DAU  05]. 
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plans  for  their  software  acquisitions,  the  ASSIP  was  shaped  to  address  those  requirements  as 
well  [PL  02], 

Organizationally,  two  main  groups  compose  the  ASSIP  The  Senior  Steering  Group  (SSG), 
comprised  of  the  PEOs,  the  Military  Deputy  (MILDEP)  to  the  ASA(  ALT),  and  the  Director 
of  the  SEI,  provides  overall  guidance  to  the  effort.  The  ASSIP  Action  Group  (AAG),  consist¬ 
ing  of  representatives  from  each  of  the  PEOs  and  from  the  Army  Software  Engineering  Cen¬ 
ters,  as  well  as  SEI  technical  staff  members,  develops  and  implements  improvement  strate¬ 
gies.  The  ASSIP  is  a  partnership  and  the  SSG  and  AAG  are  co-chaired  by  the  Army  and  the 
SEI.  Through  this  partnership,  the  SEI  offers  expert  guidance  on  software  acquisition  and 
process  issues,  provides  secretariat  services  to  the  SSG  and  AAG,  and  acts  as  a  catalyst  for 
change.  As  shown  in  Figure  1,  ASSIP  efforts  are  predicated  on  the  idea  that  improved  acqui¬ 
sition  practices  will  lead  to  better  systems  and  overall  results. 


Figure  1:  ASSIP  Uses  Improved  Acquisition  Processes  to  Yield  Better  Results 

As  with  any  improvement  effort,  an  understanding  of  the  baseline  state  of  Army  acquisition 
was  needed  for  the  ASSIP  to  be  successful.  While  there  was  plenty  of  anecdotal  evidence 
and  speculation  about  the  challenges  to  successful  SIS  acquisition,  there  was  very  little  hard 
evidence  about  the  need  to  focus  on  improvement.  The  SEI  set  about  gathering  the  necessary 
evidence  by  surveying  Army  acquisition  managers,  interviewing  PEOs,  and  participating  in 
direct  engagements  with  several  critical  Army  PMOs  using  the  BFI  method.  The  survey  and 
PEO  interviews  are  described  briefly  below;  the  results  of  the  BFI  engagements  are  the  sub¬ 
ject  of  the  body  of  this  report. 
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1.1.1  Survey  of  Acquisition  Managers 

The  SEI  made  an  initial  attempt  at  characterizing  the  state  of  Army  acquisition  through  a  sur¬ 
vey  of  Army  acquisition  managers.  While  the  survey,  conducted  in  2003,  proved  useful  to  set 
ASSIP  expectations  about  the  range  of  potential  problems,  it  did  not  provide  reliable  or  suffi¬ 
ciently  detailed  information  on  which  to  base  improvement  strategies.  Survey  instructions 
were  not  adequately  communicated  to  the  target  audience  to  achieve  consistency  among  re¬ 
spondents.  Many  managers  delegated  responsibility  for  completing  the  survey  to  staff  mem¬ 
bers  whose  backgrounds  varied  widely.  Kasunic  details  the  survey  results  and  their  limita¬ 
tions  in  a  technical  report  [Kasunic  04]. 

1.1.2  Program  Executive  Officer  Interviews 

To  gain  the  insights  of  the  Army’s  senior  acquisition  professionals,  the  SEI  conducted  inter¬ 
views  with  the  PEOs  from  April  2004  through  December  2004.  Interview  teams  consisted  of 
two  SEI  technical  staff  members,  with  one  member  acting  as  the  primary  interviewer  and  the 
other  acting  as  the  primary  recorder.  To  provide  consistency  across  the  interviews,  only  two 
individuals  acted  as  the  primary  interviewer  during  the  course  of  the  interviews. 

Interviews  were  conducted  using  a  standard  suite  of  questions,  which  was  divided  into  two 
sets:  primary  and  secondary.  The  set  of  primary  questions  dealt  with  the  PEOs’  overall  opin¬ 
ions  about  Army  acquisition  and  the  ways  in  which  the  ASSIP  could  help.  The  set  of  secon¬ 
dary  questions  dealt  mainly  with  specific  ASSIP  activities  and  activities  in  each  PEO’s  office. 

Although  the  interviews  were  conducted  on  a  confidential  basis,  the  SEI  aggregated  the  re¬ 
sults  in  a  non-attributable  manner  to  foster  analysis.  The  results  generated  by  using  the  BFI 
method  are  remarkably  consistent  with  those  generated  by  the  PEO  interviews.  Blanchette 
presents  a  detailed  discussion  of  the  PEO  interviews  and  results  in  a  special  report 
[Blanchette  05]. 
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2  Method  Description 


The  BFI  method  formed  the  backbone  of  the  AS  SIP  data- gathering  efforts  during  fiscal  year 
2003-2004.  It  offers  a  rarely  seen  view  of  PMOs,  largely  because  PMOs  are  busy  executing 
their  missions.  In  general,  PMOs  are  “too  lull  up”  to  stop  and  take  stock.  Even  though  par¬ 
ticipation  in  the  BFI  effort  may  have  been  perceived  by  some  as  a  drain  on  PMO  resources, 
the  PMOs  were  rewarded  with  information  from  a  perspective  that  reflects  both  breadth  and 
depth.  The  results  identify  issues  across  project  management  domains,  such  as  accounting 
and  systems,  in  relation  to  the  software  in  the  system. 

2.1  The  BFI  Method 

The  BFI  method  employs  a  series  of  structured  interviews  with  PMO  personnel  (similar  to 
many  assessment  techniques2),  but  the  focus  is  on  discovery  of  the  effectiveness  of  the  busi¬ 
ness  practices  and  processes  in  use  in  the  PMO,  rather  than  on  compliance  with  some  stan¬ 
dard  or  model.  To  provide  consistency  across  the  interviews,  a  list  of  interview  questions 
was  prepared  for  use  by  the  primary  interviewers.  The  structure  of  the  interview  questions 
reflected  that  of  the  CMMI  Acquisition  Module  (CMMI-AM),  Version  1.0  [Bernard  04].  Ap¬ 
pendix  B  of  this  report  contains  the  interview  questions. 

The  BFI  process  began  by  identifying  volunteer  PMOs  that  represent  each  of  the  PEOs.  Final 
selections  were  made  based  on  the  program’s  ACAT  level  and  the  degree  to  which  the  pro¬ 
gram’s  mission  was  of  a  software-intensive  nature.  Selected  PMOs  were  then  notified  and 
PEO/AAG  coordination  activities  started,  including  scheduling  a  week  for  on-site  work,  es¬ 
tablishing  a  list  of  people  from  the  PMO  who  would  participate  in  interviews,  and  building 
the  SEI-led  team  of  interviewers.  Usually  the  PMO  point  of  contact  and  SEI’s  BFI  Team 
Lead  prepared  the  detailed  interview  schedules  based  on  how  the  program  office  was  organ¬ 
ized.  At  least  two  weeks  before  the  on-site  portion  of  the  BFI  engagement  occurred,  the  co¬ 
ordination  phase  was  completed  by  having  the  PMO  point  of  contact  agree  to  the  plan  for  the 
engagement. 

During  the  PEO/AAG  coordination  phase,  the  AAG  members  from  each  PEO  received  the 
list  of  interview  questions  as  read-ahead  material.  This  allowed  the  PEOs  to  become  familiar 
with  the  discussion  materials  and  to  support  the  PMO  staffs  prior  to  the  BFI  interviews.  The 

2  The  basic  process  for  the  BFI  follows  many  of  the  same  requirements  established  for  the  SCAMPI 
Class  B  and  Class  C  methods.  You  can  find  additional  information  on  these  methods  in  the  hand¬ 
book  Standard  CMMf  Appraisal  Method  for  Process  Improvement  (SCAMPIsm),  Version  1.1: 
Method  Definition  Document  (CMU/SEI-2001-HB-001)  at 
http://www.sei.cmu.edU/publieations/documents/0 1  .reports/0  lhbOO  1  .html. 
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point  of  contact  for  each  PMO  also  received  a  copy  of  the  interview  questions  along  with  an 
ASSIP  overview  brief  to  help  them  prepare  the  PMO  staff. 


At  least  2  wte  ahead  -1  Weeks  0  +1  +2... 


Figure  2:  BFI  Method  Synopsis 

In  addition  to  the  read-ahead  materials,  the  PMO  could  request  an  in-brief  (Phase- 1)  meeting 
with  the  SEI’s  BFI  Team  Lead.  For  PMOs  who  selected  this  option,  the  BFI  Team  Lead  pre¬ 
sented  the  in-brief,  addressed  questions,  and  worked  with  the  PM  to  finalize  the  on-site  inter¬ 
view  schedule. 

The  BFI  information-gathering  activities  took  place  on-site.  The  first  day  of  the  on-site  visit 
included  a  presentation  about  the  BFI  process  and  ASSIP  from  the  BFI  Team  Lead  to  the  PM 
and  PMO  staff.  The  PM  or  a  member  of  the  PMO  staff  then  presented  background  informa¬ 
tion  about  the  program  including  how  long  the  PM  has  been  with  the  PMO,  data  about  the 
primary  customer/end-user  of  the  program’s  product,  and  program  progress  to  date.  This 
presentation  also  provided  insight  into  the  life  cycle  of  the  program  and  some  of  the  chal¬ 
lenges  it  encountered.  Following  the  formal  group  presentations,  one-on-one  interviews 
based  on  the  roles  and  responsibilities  of  the  PMO  staff  were  conducted. 

Interview  teams  consisted  of  three  to  four  SEI  technical  staff  members,  with  one  member 
acting  as  the  primary  interviewer  and  the  other  team  members  acting  as  listeners  and  record¬ 
ers.  All  of  the  interviews  were  conducted  under  the  non-attribution,  confidentiality  guide¬ 
lines  used  in  other  SEI  appraisal  methods.  Under  these  guidelines,  each  PMO  owns  its  indi¬ 
vidual  BFI  results.  However,  the  SEI  maintains  of  a  copy  of  the  results  for  aggregation  and 
analysis  of  broad  trends.  When  best  practices  were  discovered  during  the  BFI  process,  those 
practices  were  highlighted  in  the  results  with  the  knowledge  and  consent  of  the  PMO. 

While  the  goal  of  the  interviews  was  to  discover  how  Army  PMOs  conduct  their  program 
management  functions,  interviewers  did  not  interrupt  the  discussions  if  they  touched  on 
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broader  acquisition  issues,  which  nearly  all  of  the  interviews  did.  Software  is  but  one  of 
many  interrelated  components  of  an  acquisition  program  and  understanding  the  whole  picture 
is  critical  to  addressing  systemic  problems  rather  than  individual  symptoms.  As  issues  arose 
during  interviews,  the  SEI’s  BFI  team  encouraged  elaboration  of  the  information  to  better 
understand  the  scope  of  the  issue. 

After  the  information-gathering  activities  of  the  interviews  were  completed,  the  BFI  team 
analyzed  the  data  to  determine  if  common  threads  were  present.  When  there  were  findings 
having  a  common  thread,  the  information  was  grouped  under  a  “theme”  that  reflected  the 
common  nature  of  the  comments.  Some  findings  were  reported  to  the  PM  and  PMO  staff 
verbatim  in  the  out-brief  to  ensure  attendees  understood  the  relevance  of  the  results.  When 
recommendations  could  be  made  based  on  the  experiences  of  the  PMO  or  BFI  team  they 
were  included  in  the  out-brief.  The  BFI  team  presented  the  themes,  findings,  and  recommen¬ 
dations  to  the  PM  and  PMO  staff  during  the  final  out-brief.  As  mentioned  earlier,  the  PMO 
retains  full  ownership  of  the  out-brief  and  determines  when  and  how  its  contents  can  be 
shared. 


2.2  Benefits  of  the  BFI  Method 

For  their  investment  in  the  BFI  process,  PMOs  received  immediate  and  confidential  feedback 
about  their  current  practices  and  can  leverage  that  information  to  develop  action  plans  based 
on  their  needs.  For  example,  one  program  used  its  results  to  validate  its  ongoing  process  im¬ 
provement  initiatives.  Another  program  used  its  results  as  the  basis  for  beginning  improve¬ 
ment  activities.  Even  some  PEOs  (the  organizational  level  above  the  PMOs)  have  used  BFI 
results  to  start  their  own  improvement  work. 

BFI  results  also  provide  PMOs  with  an  opportunity  to  anonymously  comment  on  and  influ¬ 
ence  higher  level  policies,  with  the  SEI  acting  as  an  advocate.  In  addition,  the  PMOs  also 
benefit  from  continued  expert  consultation  through  an  ongoing  relationship  with  SEI  to  moni¬ 
tor  the  successes  and  shortcomings  of  their  improvement  strategies. 

BFI  results  provide  a  basis  from  which  a  number  of  other  possible  actions  may  be  taken,  be¬ 
cause  the  daily  activities  and  business  practices  of  the  program  offices  are  reflected  in  the 
information  gathered  through  the  interview  process.  The  perceptions  of  the  PMO  staff  with 
regard  to  the  external  organizational  influences  are  important  indicators  of  how  well  higher 
level  directives  can  be  interpreted  and  implemented. 

Unlike  more  robust  assessment  methods,  the  BFI  method  does  not  require  documented  sup¬ 
port  of  assertions  about  business  practices.  The  interview  results  are  taken  at  face  value  with¬ 
out  additional  research  into  the  validity  of  the  assertions.  For  example,  the  opinions  ex¬ 
pressed  regarding  the  Army’s  Software  Blocking  policy  were  not  investigated  beyond  the 
PMO’s  viewpoint  for  the  purpose  of  developing  this  report.  Thus,  work  done  outside  of  the 
PMO  to  “harmonize”  and  clarify  the  Software  Blocking  policy  is  not  reflected  in  this  report. 
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Despite  these  limitations,  the  BFI  results  provide  a  valuable  tool  for  quickly  identifying  both 
persistent  issues  and  immediate  opportunities  for  program  improvement. 


2.3  Potential  for  BFI  Method  Improvements 

As  with  any  attempt  to  provide  meaningful  information  to  decision  makers,  the  integrity  of 
the  information  is  always  at  issue.  This  issue  is  a  constant  whether  the  information  being 
conveyed  is  anecdotal,  as  is  the  case  with  the  BFI  results,  or  subject  to  even  more  stringent 
rules,  as  with  statistical  approaches.  The  BFI  method  belongs  to  the  family  of  methods  such 
as  the  Delphi  method,3  Grounded  Theory,4  or  any  of  a  number  of  similar  information¬ 
gathering  methods.  When  applying  these  methods,  the  reports  of  interviewees  are  the  primary 
means  by  which  construals  (i.e.,  translations  or  interpretations)  of  the  information  are  the 
products  of  the  effort.  These  methods  inherently  possess  certain  risks.  The  following  list  out¬ 
lines  the  kinds  of  risks  associated  with  an  anecdotal  information-gathering  method: 

•  Interviewees  may  come  to  the  experience  with  an  explicit  information-sharing  “agenda.” 

•  Interviewers  can  “lead”  interviewees  in  eliciting  responses. 

•  To  organizations  new  to  process  improvement,  the  level  of  rigor  applied  during  the  BFI 
process  may  seem  sufficient  to  approach  systemic  organizational  problems. 

•  Terms  used  as  the  subject  of  interview  questions  (e.g.,  “requirements,”  “configuration 
management”)  often  have  important  differences  of  meaning  according  the  interviewees’ 
expertise  and  context  of  work.  Researchers  may  not  detect  these  differences. 

•  Face-to-face  interviews  cannot  assess  the  extent  of  a  problem,  yet  the  strongly  held  be¬ 
liefs  and  convictions  of  interviewees  can  impress  interviewers  that  an  issue  is  (de  facto) 
“important.” 

•  Depending  on  the  structure  of  interviews,  interviewees  may  be  induced  to  share  informa¬ 
tion  according  to  the  desires  of  their  peers  or  their  leadership. 

•  All  extrapolations  of  solutions  derived  solely  from  anecdotal  data  are  at  risk  of  being 
completely  wrong. 

The  mitigations  for  these  risks  include  the  following: 

•  Treat  solution  actions  arising  from  anecdotal  information  as  experimental  in  nature  and 
small  in  the  scope  of  their  application. 


3  The  Delphi  method  has  traditionally  been  a  technique  aimed  at  building  an  agreement,  or  consensus 
about  an  opinion  or  view,  without  necessarily  having  people  meet  face  to  face,  such  as  through  sur¬ 
veys,  questionnaires,  emails  etc.  This  technique,  if  used  effectively,  can  be  highly  efficient  and  gen¬ 
erate  new  knowledge  [Wikipedia  05a]. 

4  Grounded  theory  is  a  general  research  method  for  social  sciences  developed  by  the  sociologists 
Barney  Glaser  and  Anselm  Strauss  [Wikipedia  05b].  For  more  information,  visit 

http  ://www.groundedtheory.  com/. 
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•  Use  the  anecdotal  information  as  a  body  of  “hints”  requiring  follow-up  using  more  exact¬ 
ing  methods  of  information  gathering. 

•  Cross-reference  information  gathered  with  the  anecdotal  information  using  more  precise 
means  to  increase  confidence  in  the  integrity  of  the  information  (sometimes  referred  to  as 
triangulation  of  information). 

•  Provide  clear  guidance  and  training  to  interviewers. 

•  Ensure  specific  ground  rules  are  followed  for  all  interviews. 

Thus  far,  SEI  BFI  teams  and  organizations  applying  the  BFI  results  have  followed  a  few  of 
the  stringencies  mentioned  above  as  mitigations.  Elowever,  improvements  in  training  and 
other  aspects  of  ensuring  infoimation  integrity  are  recommended  if  the  BFI  method  continues 
to  be  employed  over  an  extended  period  of  time.  Modifications  to  the  method  could  include 
the  following: 

•  required  BFI  Team  Fead  and  interviewer  training 

•  standardized  glossary  of  terms  for  use  by  both  interviewees  and  interviewers 

•  deliberate  scoping  of  the  method  to  limit  its  use  for  relatively  near-term  improvement 
targets.  This  would  ensure  that  results  from  a  BFI  process  would  not  be  applied  widely 
without  the  application  of  much  more  exacting  methods  of  information  gathering. 
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3  The  BFI  Results 


Each  of  the  PMOs  expressed  a  variety  of  opinions  and  concerns  about  the  state  of  Army  ac¬ 
quisition.  Although  they  each  cited  specific  problems  within  their  respective  domains,  they 
also  identified  some  systemic  issues.  This  report  addresses  only  the  systemic  issues.  In  areas 
where  PMOs  could  identify  recommendations  for  addressing  some  of  the  systemic  issues,  the 
recommendations  are  provided  in  this  report.  Although  BFI  results  for  individual  programs 
remain  the  property  of  the  respective  PMOs,  the  SEI  has  aggregated  the  results  in  a  non- 
attributable  manner  to  foster  analysis.  While  the  SEI  maintains  the  confidentiality  of  indi¬ 
vidual  respondents,  this  section  of  the  report  includes  quotations  that  punctuate  the  findings. 
Note  that  the  SEI  BFI  team  did  not  attempt  to  verify  the  opinions  or  assertions  expressed 
herein. 

The  BFI  results  presented  in  this  report  are  organized  by  the  themes  or  common  subjects  that 
surfaced  across  most  of  the  PMOs.  In  the  BFI  process,  themes  represent  repeated  references 
to  common  topics  or  specific  phenomena  either  within  or  across  programs.  Themes  are  iden¬ 
tified  by  the  BFI  team  during  consolidation  of  the  information  collected  during  interviews 
and  presentations.  The  following  sections  are  based  on  the  findings  of  the  12  BFI  engage¬ 
ments  conducted  through  September  2004.  They  represent  a  collection  of  all  of  the  final  on¬ 
site  briefings  provided  to  the  programs  and  program  leadership.  Consider  each  of  the  themes 
listed  below  separately,  since  each  represents  a  particular  comment  made  by  more  than  three 
of  the  PMOs.  Upon  reviewing  the  interview  responses,  several  themes  emerged: 

•  risk  management  processes  are  not  standardized 

•  acquisition  process  policy  changes  have  created  confusion 

•  contractor  oversight  is  inconsistent 

•  staff  expertise  is  mismatched  and  TSM  groups  are  understaffed 

•  policies  are  in  constant  flux 

•  career  development  does  not  take  into  account  relevant  experience 

•  Integrated  Product  Teams  are  not  executed  effectively 

The  following  sections  discuss  each  of  these  themes  in  more  detail.  The  order  of  discussion 
is  not  relevant  or  significant  and  readers  should  not  ascribe  priority  or  importance  to  any  of 
the  themes  based  on  sequence.  For  each  theme,  there  is  a  discussion  of  what  was  found  and 
what  could  be  improved.  PMO  suggestions  for  possible  improvement  are  also  provided.  Un¬ 
der  the  theme  risk  management,  the  results  highlight  an  exemplary  implementation  of  a  risk 
management  program. 
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3.1  Theme:  Risk  Management 

Risk  management  practices  are  not  standardized  and  risks  are  inconsistently  and  insuffi¬ 
ciently  tracked,  updated,  and  addressed. 

3.1.1  What  Was  Found 

Most  PMOs  apply  some  form  of  risk  management.  Unfortunately,  most  programs  do  not 
evaluate  and  mitigate  program-impacting  risks  before  they  become  schedule  and  cost  prob¬ 
lems.  Risk  management  practices  are  not  standardized  and  risks  are  not  consistently  tracked, 
updated,  and  addressed  sufficiently.  The  likelihood  of  risk  repositories  being  up-to-date  and 
reflective  of  the  actual  technical  risks  for  the  program  is  usually  very  low.  In  this  situation,  a 
PM  should  expect  “surprises”  on  a  regular  basis.  The  environment  of  fire-fighting  in  a  pro¬ 
gram  office  is  directly  related  to  some  of  these  surprises  cropping  up. 

One  area  that  any  program  we  contacted  did  not  identify  as  a  risk  is  the  personnel  rotation 
that  affects  every  PMO.  Another  area  of  risk,  budget  cuts  and  unfunded  new  work  (policy, 
directives,  war,  updates  to  standards,  etc.),  is  regularly  realized  but  very  rarely  is  mitigation 
or  contingency  planning  conducted. 

3.1.2  What  Could  Be  Improved 

The  SETs  BFI  team  made  several  improvement  recommendations  for  each  program  office 
related  to  risk  management.  However,  some  of  the  responsibility  for  improving  risk  manage¬ 
ment  practices  across  the  DA  must  be  coordinated  across  all  levels  of  the  program  structure 
(DA,  PEO,  PMO).  Improvement  recommendations  include  standardizing  the  risk  manage¬ 
ment  method  for  the  DA,  providing  tailoring  options  and  associated  training  for  PMOs,  pro¬ 
viding  risk  identification  and  mitigation  training  for  all  program  stakeholders,  and  reviewing 
the  identified  risks  regularly.  Risk  repositories  must  be  kept  up-to-date  with  associated  miti¬ 
gation  and  bum-down  plans.  Regular  reviews  and  communication  about  program  risks  should 
be  part  of  the  program  monitoring  plans.  The  use  of  quantitative  methods  to  establish  the  se¬ 
verity  and  priority  of  risks  is  needed  to  remove  subjectivity  with  regard  to  the  program  risks. 
The  entire  risk  process  then  needs  to  be  enforced  via  oversight  processes  to  assure  adherence. 

Every  PMO  should  maintain  a  risk  management  program  and  a  risk  repository  that  is  separate 
from  the  contractor’s  program  and  risk  repository.  PMO-specific  risks  should  not  be  entered 
into  the  risk  repository  for  the  product.  This  particular  recommendation  caused  confusion  in 
most  program  offices. 

The  risks  that  are  specific  to  the  development,  delivery,  and  sustainment  of  the  software  and 
system  product  being  acquired  should  be  part  of  a  common  product-oriented  risk  repository. 
The  currency  and  correctness  of  the  data  in  this  repository  is  the  combined  responsibility  of 
the  program  office  staff  and  any  contractors  that  support  the  development  effort.  The  focus  of 
this  information  should  be  on  risks  to  the  product. 


12 


CMU/SEI-2005-SR-014 


Program  office  risks,  which  should  be  tracked  and  managed  separately,  include  risks  like  re¬ 
tirement  of  the  program  manager,  loss  of  funding  due  to  Congressional  actions,  and  other 
Army-specific  risks  that  could  affect  the  program  office,  but  may  not  affect  the  success  of  the 
product. 

3.1.3  An  Exemplary  Implementation 

In  one  program  office  and  PEO,  we  found  a  particularly  robust  risk  management  program. 
Risk  management  practices  are  well  documented,  supported  by  consistent  training  and  quan¬ 
titative  evaluation  of  risks,  regularly  reviewed  with  program-level  and  senior  (PEO)  man¬ 
agement,  and  information  is  kept  up-to-date  with  regard  to  program  specific  risks.  Program 
risks  that  should  not  be  viewed  by  the  contractor  are  separate  and  tracked  in  a  spreadsheet, 
since  they  are  a  smaller  and  more  tightly  controlled  group  of  items.  The  program  risks  related 
to  the  development  of  the  product  are  kept  in  a  risk  repository  that  allows  shared  access  by 
the  program  office  staff,  contractors,  and  senior  level  management.  There  is  a  well-defined 
structure  for  initially  reporting  and  evaluating  a  risk  item,  establishing  the  severity  and  prior¬ 
ity  levels  quantitatively,  developing  risk  mitigation  plans,  and  communicating  the  status  of 
each  risk  item.  With  permission  of  the  program  office  and  PEO,  the  risk  management  process 
of  PM-Excalibur  and  PEO  AMMO  at  Picatinny  Arsenal,  N.J.  is  highlighted  for  reference  by 
other  Army  program  offices. 

3.2  Theme:  Acquisition  Process  -  Traditional  vs. 
New  Mandates 

The  acquisition  policy  changes  over  the  past  five  years  have  created  confusion  and  related 
difficulties  that  are  exacerbated  by  operational  demands  for  rapid  delivery >  of  early 
capability. 

3.2.1  What  Was  Found 

As  a  result  of  acquisition  policy  changes  over  the  past  five  years,  program  offices  consis¬ 
tently  commented  on  disconnected  and  poorly  communicated  acquisition  practices.  In  the 
context  of  daily  operations  at  the  program  office,  there  is  a  fundamental  conflict  between  re¬ 
sponding  to  immediate  warfighter  needs  and  following  the  traditional  acquisition  process. 

The  traditional  acquisition  process  is  still  used  as  the  basis  for  overall  program  planning, 
funding,  and  oversight,  despite  immediate  needs  for  rapid  fielding  in  support  of  Operation 
Iraqi  Freedom  and  Operation  Enduring  Freedom.  While  the  lighter  weight  acquisition  prac¬ 
tices  required  for  rapid  fielding  are  welcomed,  the  removal  of  levels  of  guidance  causes  some 
confusion  and  related  difficulties.  For  example,  external  organizations  that  must  review  and 
approve  deliverables  at  specific  milestones  are  not  using  a  process  that  fully  reflects 
rapid/early  fielding  or  spiral  development  methodologies. 
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3.2.2  What  Can  Be  Improved 

Improvement  opportunities  associated  with  this  theme  emphasize  the  need  to  balance  the  pol¬ 
icy-driven  formal  acquisition  process  with  the  ability  of  PMOs  to  respond  quickly  to  war¬ 
fighter  needs.  A  single  group  that  harmonizes  the  policies  and  processes  either  for  the  DA  or 
across  the  DoD  should  be  identified.  This  group  should  also  provide  support  for  implementa¬ 
tion  of  the  harmonized  policies.  Programs  piloting  new  initiatives  like  rapid  fielding  need  the 
flexibility  and  explicit  authority  to  “violate”  policies  that  conflict  with  the  pilot  effort. 

The  tension  between  rapid  delivery  and  formal  acquisition  processes  is  also  reflected  in  the 
way  requirements  are  conveyed  to  PMOs.  In  the  Army,  the  TRADOC  System  Managers 
(TSMs)  represent  the  user  community  to  the  PMO.  Some  TSMs  may  also  need  to  adapt  their 
processes  to  the  way  warfighter  needs  are  addressed  with  initiatives  such  as  the  Rapid  Field¬ 
ing  Initiative,  Early  Fielding,  and  so  on. 

3.3  Theme:  Oversight 

Oversight  and  monitoring  of  the  contractor  s  system  engineering  and  management  practices 
are  not  executed  consistently. 

3.3.1  What  Was  Found 

BFI  teams  found  that  the  PMOs’  oversight  and  monitoring  of  systems  engineering,  software 
development,  and  management  practices  of  the  responsible  organizations  is  inconsistent. 
While  many  of  the  practices  of  the  PMOs  are  very  good,  they  are  not  executed  consistently 
due  to  lack  of  documentation  and  training.  This  is  especially  true  in  cases  where  the  organi¬ 
zations  responsible  for  the  engineering  and  development  work  are  external  contractors. 

Some  program  offices  use  software  and  systems  engineering  staffs  matrixed  from  other  inter¬ 
nal  organizations,  such  as  the  Research,  Development,  and  Engineering  Command  (RDEC) 
software  engineering  centers  (SECs),  who  provide  good  management  and  monitoring  of 
software  development  practices.  Unfortunately,  this  does  not  apply  to  all  program  offices. 
This  results  in  inadequate  insight  into  the  processes  used  by  the  contractors  to  develop  or  test 
the  software  and  integrate  the  system.  Specific  aspects  of  this  finding  include  the  following: 

•  Development  documentation  and  metrics  are  not  always  available  to  the  program  office. 

•  PMOs  lack  the  experience  and  expertise  needed  to  review  development  documentation 
and  metrics. 

•  PMOs  do  not  enforce  contractors  to  adhere  to  software  development  processes. 

•  Oversight  and  management  processes  within  the  PMOs  are  ad  hoc,  inconsistent,  and  un¬ 
documented. 

•  No  common  program  monitoring  and  review  processes  exist  (between  the  PMOs  and 
contractors  and  between  the  prime  contractor  and  subcontractors). 
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•  PMOs  have  multiple  reporting  chains  that  require  similar  information  to  be  maintained  in 
different  formats.  As  a  result,  the  amount  of  time  that  the  program  office  spends  prepar¬ 
ing  information  for  oversight  is  excessive  or  redundant. 

3.3.2  What  Can  Be  Improved 

PMs  should  demand  sufficient  staff  who  possess  the  software  and  systems  engineering  exper¬ 
tise  needed  to  define  the  required  metrics  for  the  program  and  objectively  evaluate  software 
development  processes.  Staffs  should  have  training  and  development  experience.  Each  PMO 
should  implement  an  oversight  and  management  process  to  coordinate  and  control  internal 
administrative  information  (IPT  status,  metrics,  schedules,  etc.).  This  process  should  include 
defining  and  documenting  current  practices  (IPT  start-up,  mentoring,  document  review,  re¬ 
quirements  writing  and  verification,  etc.).  After  processes  are  documented,  PMOs  should  es¬ 
tablish  repositories  for  storing  metrics,  lessons  learned,  templates,  historical  data,  and  so  on. 
Regularly  reviewing  the  information  in  the  repository  allows  the  PMO  to  reuse  the  informa¬ 
tion  that  was  successful  and  improve  the  areas  that  need  attention. 

Across  the  Army,  there  is  a  need  to  develop  training  and  mentoring  programs  for  new  PMs 
and  mechanisms  to  share  best  practices  across  PMs. 

3.4  Theme:  Staff  Skills 

Program  management  offices  do  not  have  personnel  with  specialized  skills  to  respond  to  all 
of  the  demands  of  their  jobs.  In  addition,  TSM  groups  are  understaffed. 

3.4.1  What  Was  Found 

PMOs  are  not  sufficiently  staffed  with  personnel  who  have  the  appropriate  specialized  skills 
to  respond  to  all  of  the  demands  of  their  jobs.  In  addition  to  PMO  staffing  issues,  TSM 
groups  are  often  understaffed  and  struggle  to  represent  users  effectively.  Key  skills  are  being 
lost  from  the  both  TSM  groups  and  Army  Acquisition  Corps,  making  continuity  of  end-user 
knowledge  and  management  of  contractors  difficult.  Less  experienced  staffs  do  not  have  the 
backgrounds  needed  to  effectively  monitor  contractors  in  a  less  restrictive  contracting  envi¬ 
ronment.  Existing  staffs  cannot  adequately  evaluate  reuse  claims,  requirements  documenta¬ 
tion,  architecture,  detailed  design,  progress  at  formal  technical  reviews,  or  articulate  the  soft¬ 
ware  risks  on  the  program.  In  addition,  there  is  not  enough  software  expertise  available  to 
manage  programs  adequately. 

Some  program  offices  do  an  excellent  job  of  leveraging  external  assets  available  to  them,  like 
SECs,  to  meet  program  needs.  In  other  program  offices,  staffing  limitations  force  PMs  to  use 
support  contractors  or  their  prime  contractor  for  program  office  support.  While  support  con¬ 
tractors  have  been  integrated  successfully  into  many  PMOs  (especially  when  the  support  con¬ 
tractors  retired  from  the  organization  for  which  they  are  now  working),  not  all  of  the  program 
offices  have  access  to  or  funding  for  support  contractors. 
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3.4.2  What  Can  Be  Improved 

The  SEI  BFI  team  recommends  that  PMOs  investigate  existing  program-level  and  manager 
practices  and  lessons  learned  and  take  advantage  of  the  services  provided  by  Army  organiza¬ 
tions  like  the  SECs,  Defense  Contract  Management  Agency  (DCMA),  and  the  Cost  and  Eco¬ 
nomic  Analysis  Center  (CEAC).  The  CEAC  is  one  of  the  groups  that  provide  essential  exper¬ 
tise  to  the  program  offices.  The  measurement  expertise  of  the  CEAC  group  should  be 
consistently  employed  to  guide  programs,  especially  to  help  programs  support  estimates  with 
historical  data.  Use  the  knowledge  on  existing  programs  to  build  a  forum  for  PMs  to  discuss 
program  management  issues  and  solutions.  Consider  using  subject  matter  experts  from  or¬ 
ganizations  like  Federally  Funded  Research  and  Development  Centers  (FFRDCs)  like 
MITRE  and  the  SEI. 

There  also  needs  to  be  a  better  balance  between  the  number  of  people  who  work  for  the  gov¬ 
ernment  and  those  contracted  to  support  the  PMO,  with  the  preferred  emphasis  on  govern¬ 
ment  staff  (military  and/or  civilian).  More  than  one  PM  emphasized  the  importance  of  ensur¬ 
ing  that  the  warfighter  mission  stays  at  the  top  of  the  program’s  priority  list.  To  this  PM,  the 
only  way  to  keep  the  priority  of  the  warfighter  mission  at  the  correct  level  is  to  make  sure 
that  the  PM  or  Deputy  PM  is  a  “green  suiter”  (  Army  officer).  The  insistence  to  use  Army 
military  or  civilian  personnel  also  reflects  the  need  for  people  who  have  both  domain-specific 
skills  and  knowledge  of  the  Army.  One  PM  specifically  cited  non-military  staff  not  under¬ 
standing  the  culture  and  the  associated  inability  to  address  the  hierarchy  of  the  Army  as  a 
problem  area.  On  the  other  hand,  the  continuity  that  PMOs  find  through  the  use  of  non¬ 
military  deputies  or  technical  directors  is  also  needed  to  ensure  that  program  and  contract 
knowledge  is  bridged  when  PMs  leave  for  their  next  military  assignment  or  retire. 

3.4.3  Suggested  Improvement 

The  issue  of  command  continuity  generated  a  few  ideas  for  improvement  among  participants. 

•  Develop  a  “scorecard”  for  the  life  of  a  program,  so  that  a  PM  continues  to  bear  some  re¬ 
sponsibility  for  program  success  even  after  moving  to  a  new  assignment. 

•  Make  assignment  durations  flexible,  so  that  a  PM  remains  in  a  position  until  achieving 
some  measurable  accomplishment. 

•  Shorten  approval  cycles  by  restructuring  the  leadership  group  to  ensure  only  decision 
makers  are  in  the  approval  loop. 

3.5  Theme:  Policy 

Program  management  offices  perceive  OSD  and  Department  of  the  Army  policy  and  direc¬ 
tives  as  being  in  constant  flux,  making  it  difficult  to  identify  or  develop  policy  interpretation 
expertise. 
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3.5.1  What  Was  Found 


Program  offices  report  that  they  perceive  policies  and  regulations  as  being  in  constant  flux, 
making  it  difficult  to  identify  or  develop  policy  inteipretation  expertise.  It  seems  to  the  PMOs 
that  the  current  practice  is  to  develop  policies  in  stovepipes.  When  a  policy  or  directive 
reaches  the  PMO,  there  is  little  or  no  guidance  for  implementation.  In  many  cases,  policy 
changes  are  mandated  without  funding  or  schedule  adjustments  being  made  for  program  im¬ 
plementation.  In  some  cases,  the  policy  implementation  is  retroactive  to  the  beginning  phases 
of  a  program.  This  may  force  a  program  to  redo  months  or  years  of  work.  The  application  of 
new  or  changed  policies  for  ongoing  programs  affects  the  entire  program  profile  (funding, 
requirements,  schedule,  resources,  etc.). 

Interviewees  often  spoke  about  the  need  for  their  questions  about  policy  and  regulations  to  be 
answered  in  a  consistent,  timely,  and  accurate  fashion.  For  example,  Joint  Technical  Architec¬ 
ture-Army  (JTA-A)  and  Software  Blocking  documentation  is  not  comprehensive  and  should 
be  replaced  with  a  more  thorough,  but  concise  standard.  Further,  particular  elements  of  the 
JTA-A  and  the  Software  Blocking  policy  appear  to  be  at  odds  with  each  other.  In  addition, 
following  the  JTA-A  forces  system  architectures  to  be  more  restricted  and  closed  compared  to 
pure  JTA  implementations.  Their  superiors  or  their  peers  do  no  communicate  key  knowledge 
about  JTA-A  and  Software  Blocking  issues  effectively  to  the  PMOs. 

In  other  cases,  it  is  unclear  to  program  offices  which  version  of  the  policies  to  use.  Even  the 
documents  considered  to  be  the  “definitive  source”  of  the  policy  are  missing  appropriate  date 
and  version  numbers  to  indicate  currency.  This  lack  of  configuration  control  causes  imple¬ 
mentation  delays  and  may  require  PMOs  to  perform  multiple  reviews  to  determine  which 
documents  are  complete  and  correct. 

3.5.2  What  Can  Be  Improved 

Interviewees  suggest  that  there  needs  to  be  a  process  by  which  policies  and  regulations  are 
analyzed  for  impact  on  PMOs  before  they  are  implemented.  An  example  of  this  would  in¬ 
clude  getting  different  programs  to  agreement  on  standards  for  the  types  and  levels  of  inter¬ 
operability  needed  between  them.  This  ensures  that  it  is  then  possible  to  assess  whether  or  not 
interoperability  is  achieved. 

The  authors  of  policies  and  regulations  should  be  accountable  for  addressing  implementation 
and  interpretation  issues.  The  current  resources  available  to  programs  to  address  such  ques¬ 
tions  are  viewed  as  ineffective.  Another  suggestion  that  surfaces  frequently  is  that  policy 
changes  should  be  supported  by  training,  consistent  and  knowledgeable  support,  interpreta¬ 
tion,  and  enforcement.  Along  these  same  lines,  new  policies  should  include  examples  (opera¬ 
tional  examples  of  implemented  policy)  where  applicable. 

Additionally,  the  Army  Acquisition  Policy  (AR  70-1)  is  the  Army’s  top-level  acquisition  pol¬ 
icy.  However,  the  systems  and  software  policies  specified  in  this  document  do  not  pertain  to 
IT  organizations  that  develop  systems  for  the  Army.  Many  “new”  systems  products  being 
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developed  use  commercial  off-the-shelf  (COTS)  products  and  commercial  IT-oriented  prac¬ 
tices  that  fall  outside  of  the  “weapon  systems”  orientation  of  Army  policies.  This  gap  needs 
to  be  addressed  as  part  of  the  overall  interoperability  and  systems  architecture  policy  solution 
for  the  Army. 

3.5.3  Suggested  Improvement 

A  seemingly  simple  suggestion  from  one  PMO  is  to  have  OUSD(  AT&L)  policy  writers  coor¬ 
dinate  with  service  counterparts  to  provide  consistent  acquisition  policies  throughout  DoD. 

3.6  Theme:  Career  Development 

Program  management  certification  does  not  sufficiently  recognize  the  value  of  developmen¬ 
tal,  operational,  TSM,  DA,  or  OSD  assignments. 

3.6.1  What  We  Found 

DoD  program  management  certification  standards  do  not  recognize  practical  experience  as  a 
contributor  to  a  PM’s  success.  Becoming  an  effective  program  or  product  manager  requires 
having  the  right  developmental  assignments.  PMs  who  had  operational,  OSD,  Pentagon,  or 
TSM  assignments  prior  to  taking  on  PM  responsibilities  appear  to  be  more  successful. 

3.6.2  What  Can  Be  Improved 

Balance  staff  rotations  at  senior  levels  with  changes  to  the  overall  Army  mission  and  needs. 
Prospective  PMs  appear  to  benefit  from  spending  an  “internship”  period  in  ASA(ALT),  DA, 
or  OSD  offices.  In  addition,  broad  exposure  to  the  requirements  development  process  was 
cited  as  beneficial  to  prospective  PMs.  Interviewees  also  felt  that  TRADOC  staff  should  par¬ 
ticipate  in  an  internship  at  a  PMO.  When  these  assignments  are  not  possible,  PMs  and  Army 
civilians  with  Army  staff  experience  and  technical  and  domain  program  experience  should 
mentor  acquisition  coips  members  who  plan  to  become  PMs. 

3.6.3  Suggested  Improvement 

Interviewees  identified  several  ideas  for  improving  the  PM  selection  process: 

•  Allow  PM  candidates  to  bid  for  positions  for  which  their  qualifications  are  appropriate. 
Make  an  applicant’s  preference  statement  meaningful  by  giving  it  due  consideration.  Fo¬ 
cus  on  getting  the  right  person  for  the  job. 

•  Let  candidates  work  together  to  select  the  jobs  they  want  (and  for  which  they  possess  the 
right  qualifications)  and  negotiate  with  each  other  to  determine  final  assignments. 

•  Do  not  assign  “take  it  or  leave  it”  positions,  which  force  too  many  PMs  to  end  their  Army 
careers. 
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3.7  Theme:  Integrated  Product  Teams 

Some  programs  do  not  execute  Integrated  Product  Teams  effectively. 

3.7.1  What  Was  Found 

The  good  news  in  this  area  is  that  all  programs  use  IPTs  or  working  groups  to  develop  work 
products  and  manage  programs.  Most  of  these  teams  facilitate  good  communications  between 
members  of  the  development  team  and  PMO  staff. 

On  the  other  hand,  some  IPTs  are  not  executed  in  a  manner  consistent  with  the  DoD’s  vision 
of  an  Integrated  Product  and  Process  Development  approach.  For  example,  in  some  IPTs,  not 
all  disciplines  are  represented,  no  common  set  of  processes  is  defined,  there  is  no  shared  vi¬ 
sion  of  the  product  objectives,  or  there  are  no  clearly  defined  roles  and  responsibilities.  This 
finding  reflects  the  danger  of  implementing  a  mandate  without  the  associated  training  and 
tools. 

3.7.2  What  Can  Be  Improved 

Strengthen  IPT  practices  by  ensuring  consistent  and  up-to-date  training  for  team  members 
and  managers.  Produce  charters  for  each  team  that  defines  team  membership,  the  vision  or 
mission  of  each  team,  roles  that  each  team  must  perform,  responsibilities,  reporting  and  esca¬ 
lation  processes,  and  how  improvements  will  be  made  to  the  products  and  team  structure. 

Do  not  force  PMOs  to  adopt  an  IPT  approach.  When  an  IPT  approach  is  not  appropriate  for 
the  work  or  business  structure,  allow  the  use  of  working  groups  or  other  organizational  struc¬ 
tures. 
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4  Next  Steps 


This  report  documents  the  results  of  the  BFI  activities  conducted  as  part  of  AS  SIP  data- 
gathering  efforts.  BFI  activities  are  one  of  several  efforts  geared  toward  creating  an  accurate 
picture  of  processes  supporting  Army  software  and  systems  acquisition.  The  BFI  interviews 
of  PMO  staffs  operate  in  conjunction  with  (but  do  not  depend  on)  other  activities  such  as  the 
PEO  survey  and  the  PEO  interviews  described  in  Section  1.1  of  this  report.  Our  goal  is  to 
develop  an  “accurate  picture”  for  the  purposes  of  making  high-leverage  changes  that  benefit 
individual  programs  and  Army  acquisition  in  general. 


4.1  Analyzing  the  Findings 

One  of  the  most  important  next  steps  is  to  analyze  the  BFI  results,  along  with  the  results  of 
the  other  data-gathering  efforts  discussed  in  Section  1 . 1 ,  to  discover  which  problems  are  the 
most  troublesome  systemically.  Identifying  those  problems  reveals  potential  high-impact 
improvement  opportunities. 
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Figure  3:  Analysis  of  BFI  Findings 

Figure  3  depicts  the  raw  findings  from  the  12  PMOs  that  participated  in  the  BFI  process.  The 
numbers  of  findings  are  denoted  by  the  numeric  values,  which  are  categorized  according  to 
the  process  areas  identified  in  the  Capability  Maturity  Model  Integration  Acquisition  Module 
(CMMI-AM),  Version  1.0  [Bernard  04].  As  might  be  expected  for  PMOs,  this  analysis  shows 
that  the  findings  are  overwhelmingly  in  the  Project  Planning  and  Project  Monitoring  and 
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Control  process  areas.  Focusing  improvement  efforts  in  these  areas  is  likely  to  result  in  a 
high  return  on  investment  for  PMOs. 


4.2  Using  the  Findings 

The  results  of  the  BFI  activities  suggest  that  there  are  several  avenues  of  pursuit  for  improv¬ 
ing  Army  acquisition,  especially  as  it  relates  to  software-intensive  systems.  While  each  pro¬ 
gram  office  had  concerns  specifically  related  to  its  own  domain,  a  number  of  common  issues 
arose.  Common  issues  revolved  around  skills  and  training  of  PMs,  policy  and  its  application, 
the  acquisition  organization  and  its  workings,  and  the  use  of  processes  and  their  effects. 

The  AAG  updates  its  ASSIP  Strategic  Software  Improvement  Master  Plan  (SSIMP)  each  fis¬ 
cal  year.  PEOs  approve  the  SSIMP,  which  is  then  signed  by  the  MILDEP.  The  BFI  results 
have  helped  to  guide  the  initiatives  included  in  the  FY06  SSIMP,  which  defines  specific  tasks 
to  alleviate  some  of  the  perceived  problems  in  SIS  acquisition.  Tasks  are  organized  within 
initiatives  in  the  following  areas: 

•  Acquisition  Process  Improvement 

•  Education 

•  Measurement 

•  Architecture 

•  System  of  Systems  Integration 

•  Management,  Analysis,  and  Strategic  Planning 

As  the  ASSIP  continues  to  promote  acquisition  process  improvement,  the  SEI  will  revisit  the 
BFI  results  to  determine  how  process  changes  have  affected  Army  SIS  acquisition.  The  SEI 
will  also  broaden  the  pool  of  interviewees  to  include  other  senior  stakeholders  in  Army  ac¬ 
quisition. 

4.3  Issues  Related  to  Information  Processing 

As  discussed  in  Section  2,  the  BFI  method  is  a  non-parametric  form  of  information  gathering. 
This  implies  that  the  opinions  solicited  during  interviews  may  require  further  investigation  to 
substantiate  the  accuracy  and  character  of  the  information  prior  to  taking  actions  towards  im¬ 
provement. 

As  programs  determine  courses  of  action,  it  may  be  necessary  to  conduct  more  focused  and 
refined  information-gathering  efforts  that  have  greater  quantitative  precision  in  order  to 
evaluate  improvement  options.  One  alternative  for  collecting  detailed  information  is  the  use 
of  model-based  appraisals.  A  model-based  appraisal  provides  more  precision  and  a  method 
for  benchmarking  the  program  against  an  industry  and  government  measurement  mechanism. 
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Often,  the  use  of  metrics  can  help  to  discern  problems  that  may  be  more  pressing  but  less 
visible. 
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5  Summary 


Studies  on  the  return  on  investment  for  process  improvement  efforts  are  hopeful.  By  using 
disciplined  methods  of  intervention,  identifying  opportunities  for  improvement,  and  defining 
clear  improvement  objectives,  programs  can  achieve  high  rewards  as  they  seek  to  improve 
processes  [Capell  04].  Participating  in  BFI  activities  is  a  good  starting  point  for  a  program’s 
improvement  efforts. 

Application  of  the  BFI  method  fills  a  gap  in  information-gathering  methods  for  the  AS  SIP. 
While  the  BFI  method  is  not  without  flaws,  it  provides  a  flexible,  readily  applicable  means  of 
acquiring  information  quickly.  Results  from  the  compilation  of  all  of  the  BFI  engagements 
conducted  to  date  show  that  there  are  opportunities  for  improvement  within  the  DA.  The  re¬ 
sults  also  show  that  important  information  exists  within  the  program  offices  that  form  the 
foundation  for  those  improvements. 

While  the  PMOs  expressed  a  great  many  concerns  and  frustrations  with  the  current  state  of 
Army  SIS  acquisition,  they  also  offered  some  thought-provoking  ideas  for  improvement. 
More  work  remains  to  understand  the  root  causes  of  problems  within  SIS  acquisition  and 
tackling  the  problems  requires  a  long-term  commitment.  The  ASSIP  provides  the  impetus 
and  infrastructure  to  identify  these  issues  and  formulate  strategic  approaches  to  address  them. 
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Feedback 


Through  its  Acquisition  Support  Program,  the  SEI  is  working  to  help  improve  SIS  acquisition 
across  the  U.S.  government.  Consequently,  the  SEI  is  very  interested  in  hearing  how  the  per¬ 
spectives  presented  here  compare  with  those  of  senior  acquisition  officials  in  the  other  ser¬ 
vices,  defense  agencies,  and  other  federal  agencies. 

Please  send  questions  or  comments  about  this  report  to  Kristi  L.  Keeler 
(kkeeler@sei.cmu.edu). 
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Appendix  A  Acronyms  and  Abbreviations 


The  alphabetical  listing  below  contains  all  acronyms,  abbreviations,  and  their  meanings  as 
used  in  this  report. 


AAG 

ACAT 

Ammo 

ASA(ALT) 

ASSIP 

BF1 

CEAC 

CM 

CMM 

CMMI 

CMU 

COTS 

C/PAR 

DA 

DAU 

DCMA 

DoD 

FFRDC 

FY 


ASSIP  Action  Group 
Acquisition  Category 
Ammunition 

Assistant  Secretary  of  the  Army  for  Acquisition,  Logistics,  and 
Technology 

Army  Strategic  Software  Improvement  Program 
Benchmarking  for  Improvement 
Cost  and  Economic  Analysis  Center 
Configuration  Management 
Capability  Maturity  Model 
Capability  Maturity  Model  Integration 
Carnegie  Mellon  University 
Commercial  Off-the-Shelf 
Corrective/Preventive  Action  Report 
Department  of  the  Army 
Defense  Acquisition  University 
Defense  Contract  Management  Agency 
Department  of  Defense 

Federally  Funded  Research  and  Development  Center 
Fiscal  Year 
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HQDA 

IPT 

JTA-A 

MILDEP 

OSD 

OUSD(AT&L) 

PEO 

PM 


PMO 

QFD 

RDEC 

SCAMPI 

SEC 

SEI 

SIS 

SR 

SSG 

SSIMP 

TRADOC 

TSM 


Headquarters,  Department  of  the  Army 
Integrated  Product  Team 
Joint  Technical  Architecture -Army 
Military  Deputy 

Office  of  the  Secretary  of  Defense 

Office  of  the  Undersecretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics 

Program  Executive  Officer 
Program  Executive  Office 
Project  Manager 
Program  Manager 
Product  Manager 
Program  Management  Office 
Quality  Function  Deployment 

Research,  Development,  and  Engineering  Command 

Standard  CMMI  Appraisal  Method  for  Process  Improvement 

Software  Engineering  Center 

Systems  Engineering  Center 

Software  Engineering  Institute 

Software-Intensive  Systems 

Special  Report 

Senior  Steering  Group 

Strategic  Software  Improvement  Master  Plan 
Training  and  Doctrine  Command 
TRADOC  System  Manager 
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Appendix  B  Benchmarking  for 

Improvement  Interview 
Questions 


This  appendix  documents  the  questions  and  comments  used  for  conducting  interviews  of 
Army  software-intensive  system  acquisition  stakeholders,  subject  matter  experts,  and  pro¬ 
gram  members  both  within  the  program  participating  in  the  BFI  process  and  in  associated 
organizations. 

The  objective  of  the  interview  questions  is  to 

•  determine  what  topics  interviewees  believe  are  important  to  cover  for  benchmarking 

•  determine  the  objective  view  of  the  interviewee  regarding  priority  problems,  risks,  issues, 
opportunities  and  best  practices  most  important  to  the  program 

•  obtain  interviewees’  suggestions  as  to  the  kind  of  benchmarks  they  think  program  prac¬ 
tices  should  be  evaluated  against 

Setting  the  Stage  for  Questions 

This  work  is  part  of  an  effort  to  produce  an  Army-wide  software  intensive  system  Acquisition 
Improvement  Master  Plan  to  identify  problem  areas  and  opportunities  where  significant  im¬ 
provement  can  take  place  and  to  work  with  selected  programs  to  provide  exemplars  on  how 
innovations  have  been  or  could  be  achieved. 

Our  first  step  is  to  get  to  know  the  program  we  are  working  with  through  interviewing  a  cross 
section  of  program  members  and  stakeholders. 

We  will  start  each  interview  by  asking  each  interviewee  some  very  general  and  open-ended 
questions,  proceed  with  follow-up  questions  for  clarification  and  elaboration  and  finally,  and 
if  there  is  time,  ask  questions  on  various  topics  in  software  intensive  system  acquisition  that 
have  been  identified  by  previous  investigation  as  areas  of  interest. 
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Interview  Questions 


General 


Ql.  Could  you  please  describe  your  experience  with  the  program, 

Experience,  Role,  and  what  your  role  is,  and  what  areas  of  work  you  think  are  exem- 
Areas  of  Work  plary  and/or  in  need  of  improvement. 


Comments 


Probing  Questions 


There  are  various  ways  that  the  respondent  might  have  knowledge 

of  the  program  and  it  would  be  helpful  to  know  how  this  experience 

was  acquired.  Examples  include: 

•  Previous  work  as  part  of  a  similar  acquisition  configuration 
management  and  testing  program  or  system  of  systems  pro¬ 
gram;  if  so,  what  was  their  role?  (PEO?  PM?  Team  member?  In 
the  overall  system  program,  in  one  of  the  systems  programs?) 

•  Policy-maker  covering  acquisition  configuration  management 
and  testing  programs  and/or  system  of  systems  programs 

•  Research  and/or  collaboration  (in  the  acquisition  configuration 
management  and  testing  and/or  interoperability  arena) 

•  A  developer  who  worked  for  an  acquirer  and  has  knowledge  of 
acquisition  from  the  configuration  management  and  testing 
and/or  interoperability  perspective 

The  purpose  of  Q 1  is  to  address  these  issues  and  understand  the 

perspective  and  context  from  which  the  respondent  is  speaking. 

•  Given  those  experiences,  what  were  the  primary  problems  you 
encountered?  Please  provide  the  context  for  each  problem  that 
is  described. 

•  What  were  the  key  factors  that  contributed  to  successes  in  your 
previous  acquisition  work?  Please  provide  the  context  for  each 
success  that  is  described. 

•  The  term  “key  factors”  can  be  anything  including  policy,  pro¬ 
cedure,  management,  etc. 

•  If  you  could  change  anything  with  respect  to  the  program,  what 
would  it  be? 


32 


CMU/SEI-2005-SR-014 


Questions  Based  on  the  CMMI  Acquisition  Module  (CMMI-AM) 

Questions  Q2.l-Q2.16  are  based  on  the  CMMI  Acquisition  Module  (CMMI-AM),  Version  1.0. 
Figure  4  contains  the  process  areas  in  the  CMMI  Acquisition  Module.  These  process  areas 
serve  as  the  basis  for  forming  questions  exploring  potential  sources  of  “best  practices”  in  and 
across  acquisition  programs.  With  a  clear  goal,  or  “picture  of  success,”  a  program  can  identify 
a  wide  range  of  enablers  and/or  obstacles  to  achieving  their  goal  using  the  process  areas  as  a 
starting  point  during  the  interview  sessions.  Programs  that  share  similar  goals  might  benefit 
from  emulating  practices  that  have  been  found  to  be  successful  or  that  successfully  address 
similar  difficulties. 


The  execution  and  institutionalization  of  a  practice  may  indicate  a  candidate  best  or  exemplary 
practice.  For  example,  use  of  integrated  product  teaming  to  address  system  of  systems  pro¬ 
gram  management  may  indicate  a  whole  host  of  candidate  best  or  exemplary  practices  that 
may  be  applicable  in  other  program  contexts.  Careful  documentation  and/or  refinement  of  the 
practice  would  not  only  help  the  given  program  but  other  programs  as  well. 

On  the  other  hand,  the  lack  of  a  practice,  or  insufficiency  in  performing  a  practice  could  indi¬ 
cate  an  area  for  which  an  improvement  goal  needs  to  be  set  and  steps  taken  toward  achieving 
it.  For  example,  if  a  program  does  not  trace  user  requirements  from  user  capability  statements 
or  operational  needs  down  to  the  product  components  that  implement  those  capabilities  or 
needs,  this  insufficiency  should  cause  concern  about  the  program’s  ability  to  verify  or  validate 
the  system  to  their  user's  satisfaction.  Addressing  this  insufficiency  would  be  beneficial  to  the 
program.  Documenting  the  insufficiency  and  the  steps  being  taken  to  address  it  might  also  be 
useful  to  other  programs  that  may  be  facing  the  same  problem. 


Once  enough  descriptions  of  exemplary  practices  have  been  generated,  candidate  best  practice 
areas  will  emerge.  They  can  become  the  basis  for  improvement  planning  efforts  across  the 
Army. 


•  Project  Planning 

•  Project  Monitor  and 
Control 

•  Integrated  Project 
Management 

•  Risk  Management 

•  Integrated  Teaming 

•  Solicitation  and  Contract 
Monitoring 


•  Requirements  Management 

•  Requirements  Development 

•  Verification 

•  Validation 


•  Configuration  Management 

•  Process  and  Product  Quality 
Assurance 

•  Measurement  and  Analysis 

•  Decision  Analysis  and 
Resolution 

•  T ransition  to  Operations  and 
Support 

•  Organizational  Environment 
for  Integration 


Figure  4:  CMMI-AM  Process  Areas 
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Q2.1 

Program  Planning 


How  does  your  program  establish  and  maintain  plans  that  de¬ 
fine  the  program  activities? 


Probing  Questions 


CMMI  Practices 


•  How  does  your  program  establish  the  planning  parameters 
(e.g.,  guidelines  and  constraints)  for  estimating  effort  and  cost 
of  its  development  activities?  Is  historical  data  used? 

•  How  does  your  program  establish  and  maintain  its  plans? 

•  How  does  your  program  obtain  commitments  to  the  program 
plan?  How  are  these  maintained? 

•  How  does  your  program  ensure  that  program  planning  is  insti¬ 
tutionalized  as  a  managed  process? 

•  How  does  your  program  perform  replanning? 

•  How  does  your  program  plan  for  the  identification  and  analysis 
of  risk? 

Does  the  program  perform  any  of  the  following  practices?  Does  the 
program  perform  any  particularly  well?  Are  their  some  the  program 
needs  or  wants  to  improve? 

•  Estimate  the  Scope  of  the  Program 

•  Establish  Estimates  of  Program  Attributes 

•  Define  Program  Life  Cycle 

•  Determine  Estimates  of  Effort  and  Cost 

•  Establish  the  Budget  and  Schedule 

•  Identify  Program  Risks 

•  Plan  for  Data  Management 

•  Plan  for  Program  Resources 

•  Plan  for  Needed  Knowledge  and  Skills 

•  Plan  Stakeholder  Involvement 

•  Establish  the  Program  Plan 

•  Review  Subordinate  Plans 

•  Reconcile  Work  and  Resource  Levels 

•  Obtain  Plan  Commitment 
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Additional  Topics 


Potential  sources  of  exemplary  practices,  difficulties,  or  lacks 


□  Project  plans: 

□  Acquisition  Strategy 

□  Transition,  etc. 

□  Estimation  models/results/BOE 

□  Evidence  of  adequate  resources  and  tools 

□  Project  schedules  and/or  dependencies 

□  Management  training  for  planning 

□  People  assigned  responsibility  and  trained 

□  Risk  identification/analysis 

□  Task  planning 

□  Coordinate  with  stakeholders 

□  WBS 
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Q2.2 

Project  Monitoring 
and  Control 

Probing  Questions 


CMMI  Practices 


Additional  Topics 


How  does  your  program  establish  and  maintain  adequate  visi¬ 
bility  into  the  program’s  progress  so  that  corrective  actions  can 
be  taken  when  there  are  significant  deviations  from  the  plan? 

•  How  does  your  program  track  actual  performance  of  the  pro¬ 
gram  against  plans? 

•  How  does  your  program  plan  for  and  take  corrective  actions 
when  the  program’s  performance  or  results  deviate  significantly 
from  the  plan? 

•  How  does  your  program  ensure  that  program  monitoring  and 
control  is  institutionalized  as  a  managed  process? 

•  How  does  your  program  monitor  and  record  risks  and  risk  ac¬ 
tivities? 

•  Does  your  program  revise  the  program  plan  to  reflect  accom¬ 
plishments,  progress,  changes,  and  corrective  actions  as  appro¬ 
priate? 

Does  the  program  perform  any  of  the  following  practices?  Does  the 
program  perform  any  particularly  well?  Are  their  some  the  program 
needs  or  wants  to  improve? 

•  Monitor  Project  Planning  Parameters 

•  Monitor  Commitments 

•  Monitor  Project  Risks 

•  Monitor  Data  Management 

•  Monitor  Stakeholder  Interactions 

•  Conduct  Progress  Reviews 

•  Conduct  Milestone  Reviews 

•  Analyze  Issues 

•  Take  Corrective  Actions 

•  Manage  Corrective  Actions 

Potential  sources  of  exemplary  practices,  difficulties,  or  lacks 

□  Revised  program  plans 

□  Evidence  of  adequate  resources  and  tools 

□  Record  of  critical  facility  usage 

□  Status  reports 

□  Action  items 

□  People  assigned  responsibility  and  trained 

□  Records  of  commitment  reviews 

□  Corrective  action  plan 

□  Documented  program  and/or  milestone  reviews 
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Q2.3 

Integrated  Program 
Management 


Probing  Questions 


CMMI  Practices 


How  does  your  program  ensure  that  it  has  established  and  is 
managing  the  program  and  the  involvement  of  the  relevant 
stakeholders  according  to  an  integrated  and  defined  process 
that  is  tailored  from  the  organization’s  set  of  standard  proc¬ 
esses? 

•  How  does  your  program  ensure  that  it  uses  a  defined  process 
that  is  tailored  from  the  organization’s  set  of  standard  proc¬ 
esses? 

•  How  does  your  program  ensure  that  coordination  and  collabo¬ 
ration  with  relevant  stakeholders  is  conducted? 

•  How  are  technical  activities  coordinated  with  other  program 
and  organizational  functions? 

•  How  does  your  program  ensure  that  integrated  program  man¬ 
agement  is  institutionalized  as  a  defined  process? 

•  How  does  your  program  create  a  shared  product  vision  for  the 
program? 

•  How  are  integrated  teams  identified,  defined,  structured,  and 
tasked? 

•  How  were  your  program  plans  developed  to  include  organiza¬ 
tional  processes? 

•  How  does  your  program  ensure  that  all  appropriate  resources 
are  included  in  identifying,  negotiating,  and  tracking  critical 
dependencies? 

•  How  are  technical  issues  resolved  among  all  program  func¬ 
tions? 

Does  the  program  perform  any  of  the  following  practices?  Does  the 
program  perform  any  particularly  well?  Are  their  some  the  program 
needs  or  wants  to  improve? 

•  Establish  the  Project’s  Defined  Process 

•  Use  Organizational  Assets  for  Planning  Project  Activities 

•  Integrate  Plans 

•  Manage  Project  Using  Integrated  Plans 

•  Contribute  to  Organizational  Process  Assets 

•  Manage  Stakeholder  Involvement 

•  Manage  Dependencies 

•  Resolve  Coordination  Issues 

•  Define  the  Project’s  Shared  Vision  Context 

•  Establish  the  Project’s  Shared  Vision 
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Additional  Topics 


Q2.4 

Risk  Management 


Probing  Questions 


CMMI  Practices 


•  Determine  Integrated  Team  Structure  for  the  Project 

•  Develop  Preliminary  Distribution  of  Requirements  to  Inte¬ 
grated  Teams 

•  Establish  Integrated  Teams 

Potential  sources  of  exemplary  practices,  difficulties,  or  lacks 

□  Tailored  process 

□  Revised  plans  or  commitments 

□  Conflict  resolution  process 

□  Life  cycle  model 

□  Documented  internal  and  external  commitments 

□  Joint  recommendations  on  technical  issues 

□  Action  item  databases 

□  Schedules  for  collaborative  technical  activities 

□  IPTs 

How  does  your  program  identify  potential  problems  before  they 
occur,  so  that  risk  handling  activities  may  be  planned  and  invoked 
as  needed  across  the  life  cycle  to  mitigate  adverse  impacts  on 
achieving  objectives? 

•  How  does  your  program  define  the  requirements  and  strategy 
for  risk  management? 

•  How  does  your  program  determine  risk  sources  and  categories? 

•  How  does  your  program  address  risks  in  order  to  mitigate  ad¬ 
verse  impacts  on  achieving  objectives? 

•  How  does  your  program  ensure  that  risk  management  is  institu¬ 
tionalized  as  a  defined  process? 

•  How  does  your  program  identify  risk? 

•  Does  your  program  assess  the  likelihood  and  consequence  of 
each  risk?  What  is  the  process? 

•  How  does  your  program  monitor  the  status  of  each  risk  item? 

Does  the  program  perform  any  of  the  following  practices?  Does  the 
program  perform  any  particularly  well?  Are  their  some  the  program 
needs  or  wants  to  improve? 

•  Determine  Risk  Sources  and  Categories 

•  Define  Risk  Parameters 

•  Establish  a  Risk  Management  Strategy 

•  Identify  Risks 


38 


CMU/SEI-2005-SR-014 


Additional  Topics 


Q2.5 

Integrated  Teaming 
Probing  Questions 


CMMI  Practices 


•  Evaluate,  Classify,  and  Prioritize  Risks 

•  Develop  Risk  Mitigation  Plans 

•  Implement  Risk  Mitigation  Plans 

Potential  sources  of  exemplary  practices,  difficulties,  or  lacks 

□  Risk  management  and  mitigation  plans 

□  Risk  list 

□  Risk  rating  or  potential 

□  Contingency  plans 

□  Risk  prioritization 

□  Opportunity  management 

How  does  your  program  form  and  sustain  integrated  teams  for 
the  development  of  work  products? 

•  How  does  your  program  establish  team  tasks  and  resource 
needs  and  select  team  members? 

•  How  does  your  program  govern  the  activities  of  the  team? 

•  Do  integrated  teams  have  the  right  mix  of  skills  and  resources 
to  function  effectively? 

•  Do  team  members  all  share  a  common  picture  of  success? 

•  Are  team  activities,  responsibilities,  and  tasks  well  defined  and 
understood? 

•  Are  the  boundaries  and  communication  paths  between  teams 
well  understood? 

Does  the  program  perform  any  of  the  following  practices?  Does  the 
program  perform  any  particularly  well?  Are  their  some  the  program 
needs  or  wants  to  improve? 

•  Identify  Team  Tasks 

•  Identify  Needed  Knowledge  and  Skills 

•  Assign  Appropriate  Team  Members 

•  Establish  a  Shared  Vision 

•  Establish  a  Team  Charter 

•  Define  Roles  and  Responsibilities 

•  Establish  Operating  Procedures 

•  Collaborate  Among  Interfacing  Teams 
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Additional  Topics 


Potential  sources  of  exemplary  practices,  difficulties,  or  lacks 


Q2.6 

Solicitation  and 
Contract  Monitoring 


Probing  Questions 


□  Team  charter 

□  Representation  by  disciplines 

□  Clear,  written  agreements  and  commitments 

□  Clear  leader  in  charge 

□  Clear  tasks  and  deliverables 

□  Results 

How  does  your  program  prepare  a  solicitation  package  that 
identifies  the  needs  of  a  particular  acquisition,  select  a  supplier 
who  is  best  capable  of  satisfying  those  needs,  and  provide  lead¬ 
ership  throughout  the  life  of  the  acquisition  to  ensure  those 
needs  are  met? 

•  How  does  the  program  prepare  to  conduct  the  solicitation? 

•  How  does  the  program  select  suppliers  based  on  the  solicitation 
package? 

•  How  does  your  program  issue  contracts  based  on  the  needs  of 
the  acquisition  and  the  suppliers’  proposed  approaches? 

•  How  does  your  program  coordinate  with  suppliers  to  ensure  the 
contract  is  executed  properly? 

•  How  do  you  ensure  the  solicitation  complies  with  the  applica¬ 
ble  federal,  department,  and  service  acquisition  regulations  and 
policies? 

•  How  do  you  address  issues  appropriate  to  the  system  domain  or 
acquisition  environment  (e.g.,  supplier  process  evaluations,  op¬ 
erational  safety  suitability  and  effectiveness,  safety,  certifica¬ 
tions,  architecture  evaluations,  interoperability)? 

•  How  have  you  negotiated  and  incoiporated  into  the  agreement 
how  integrated  teams  are  formed  and  team  membership  is  es¬ 
tablished? 
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CMMI  Practices 


Additional  Topics 


Does  the  program  perform  any  of  the  following  practices?  Does  the 
program  perform  any  particularly  well?  Are  their  some  the  program 
needs  or  wants  to  improve? 

•  Designate  a  Selection  Official 

•  Establish  a  Solicitation  Package  and  Evaluation  Criteria 

•  Establish  Cost  and  Schedule  Estimates 

•  Validate  the  Solicitation  Package 

•  Evaluate  Proposals 

•  Use  Evaluation  Results  to  Select  Suppliers 

•  Establish  an  Understanding  of  the  Contract  and  Proposed  Ap¬ 
proach 

•  Establish  Communications  Processes  and  Procedures 

•  Monitor  Selected  Supplier  Processes 

•  Evaluate  Selected  Supplier  Work  Products 

•  Revise  the  Supplier  Agreement  or  Relationship 

Potential  sources  of  exemplary  practices,  difficulties,  or  lacks 

□  Solicitation  package 

□  Source  selection  activities 

□  Suppliers  proposed  approaches 

□  Contractual  agreements 

□  Negotiation  processes 

□  Evaluation  criteria 

□  Communication  strategies 

□  Contract  monitoring  approaches 

□  Contractual  changes 
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Q2.7 

Configuration 

Management 


Probing  Questions 


CMMI  Practices 


Additional  Topics 


How  does  your  program  establish  and  maintain  the  integrity  of 
work  products  using  configuration  identification,  configuration 
control,  configuration  status  accounting,  and  configuration  au¬ 
dits ? 

•  How  does  your  program  establish  and  maintain  baselines  of 
identified  work  products? 

•  How  does  your  program  track  and  control  changes  to  the  work 
products  under  CM  (configuration  management)? 

•  How  does  your  program  ensure  the  integrity  of  the  baselines? 

•  How  does  your  program  ensure  that  configuration  management 
is  institutionalized  as  a  managed  process? 

•  What  procedures,  tools,  and  media  has  your  program  estab¬ 
lished  for  CM/SCM? 

•  How  does  your  program  report  CM  status  and  what  audits  do 
you  perform? 

Does  the  program  perform  any  of  the  following  practices?  Does  the 
program  perform  any  particularly  well?  Are  their  some  the  program 
needs  or  wants  to  improve? 

•  Identify  Configuration  Items 

•  Establish  a  Configuration  Management  System 

•  Create  or  Release  Baselines 

•  Track  Changes 

•  Control  Changes 

•  Establish  Configuration  Management  Records 

•  Perform  Configuration  Audits 

Potential  sources  of  exemplary  practices,  difficulties,  or  lacks 

□  Controlled  artifacts 

□  CM/Configuration  Management  Plan 

□  Configuration  management  status  reports 

□  Revision  history  of  configured  items 

□  CM  responsibility  assigned 

□  Change  requests  and/or  metrics 

□  CCB  minutes 

□  CM  tool  training 
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Q2.8 

Process  and  Product 
QA 

Probing  Questions 


CMMI  Practices 


Additional  Topics 


How  does  your  program  provide  its  staff  and  management  with 
objective  insight  into  the  processes  and  associated  work  prod¬ 
ucts? 

•  How  does  your  program  objectively  verify  the  adherence  of 
work  products,  and  services  to  applicable  process  descriptions, 
standards,  and  products? 

•  How  does  your  program  ensure  that  identified  non-compliance 
issues  are  escalated  as  necessary  for  resolution? 

•  How  does  your  program  ensure  that  process  and  product  qual¬ 
ity  assurance  is  institutionalized  as  a  managed  process. 

•  What  activities  are  defined  in  your  quality  plan?  Are  they  be¬ 
ing  performed? 

•  How  does  your  program  periodically  report  results  of  process 
and  product  quality  assurance  to  those  affected? 

•  Are  program  personnel  informed  of  quality  review/audit  re¬ 
sults? 

Does  the  program  perform  any  of  the  following  practices?  Does  the 
program  perform  any  particularly  well?  Are  their  some  the  program 
needs  or  wants  to  improve? 

•  Objectively  Evaluate  Processes 

•  Objectively  Evaluate  Work  Products  and  Services 

•  Communicate  and  Ensure  Resolution  of  Noncompliance  Issues 

•  Establish  Records 

Potential  sources  of  exemplary  practices,  difficulties,  or  lacks 

□  Process  &  Product  Quality  Assurance  Plan 

□  Responsibility  assigned 

□  Training  provided 

□  Audit  records  and  reports 

□  Results/minutes  of  quality  reviews 

□  Reporting  mechanisms 

□  Corrective/preventive  action  reports  (C/PARs) 
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Q2.9 

Measurement  and 
Analysis 

Probing  Questions 


CMMI  Practices 


Additional  Topics 


How  does  your  program  develop  and  sustain  a  measurement 
capability  that  supports  management  information  needs? 


•  How  does  your  program  ensure  that  measurement  objectives 
and  practices  are  aligned  with  established  information  needs 
and  objectives? 

•  How  does  your  program  ensure  that  measurement  results  that 
address  information  needs  are  available? 

•  How  does  your  program  ensure  that  Measurement  and  Analysis 
is  institutionalized  as  a  managed  process? 

•  What  data  has  your  program  collected  and  stored? 

•  What  analysis  and  interpretation  have  your  program  per¬ 
formed? 

•  What  training  is  provided  to  your  program  to  support  meas¬ 
urement  collection  and  analysis? 

Does  the  program  perform  any  of  the  following  practices?  Does  the 
program  perform  any  particularly  well?  Are  their  some  the  program 
needs  or  wants  to  improve? 

•  Establish  Measurement  Objectives 

•  Specify  Measures 

•  Specify  Data  Collection  and  Storage  Procedures 

•  Specify  Analysis  Procedures 

•  Collect  Measurement  Data 

•  Analyze  Measurement  Data 

•  Store  Data  and  Results 

•  Communicate  Results 

Potential  sources  of  exemplary  practices,  difficulties,  or  lacks 

□  Measurement  plans  and  objectives 

□  Evidence  of  adequate  resources  and  tools 

□  People  assigned  responsibility  and  trained 

□  Measures  databases 

□  Results  of  data  integrity  tests 

□  Data  collection,  storage  procedures  and/or  tools 

□  Analysis  definitions,  procedures  and/or  tools 

□  Analysis  reports,  draft  reports,  presentations 

□  Collected  &  derived  measurement  data  sets 
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Q2.10 

Decision  Analysis  and 
Resolution 

Probing  Questions 


CMMI  Practices 


Additional  Topics 


How  does  your  program  establish  structured  decision  making 
techniques  and  criteria  for  use  in  evaluating  identified  alterna¬ 
tives? 

•  How  does  your  program  establish,  use,  and  evaluate  alterna¬ 
tives  when  making  decisions? 

•  How  does  your  program  establish  criteria  for  evaluating  alter¬ 
native  solutions  to  selected  issues? 

•  How  does  your  program  ensure  that  decision  analysis  and  reso¬ 
lution  is  institutionalized  as  a  defined  process 

•  How  does  your  program  elicit  alternative  solutions  to 
risks/issues? 

•  How  does  your  program  evaluate  and  document  assumptions? 

•  How  does  your  program  involve  the  stakeholders  in  the  selec¬ 
tion  process 

Does  the  program  perform  any  of  the  following  practices?  Does  the 
program  perform  any  particularly  well?  Are  their  some  the  program 
needs  or  wants  to  improve? 

•  Establish  and  Use  Guidelines  for  Decision  Analysis 

•  Select  Evaluation  Technique 

•  Establish  Evaluation  Criteria 

•  Identify  Proposed  Alternatives 

•  Evaluate  Alternative  Solutions 

•  Select  Solutions 

Potential  sources  of  exemplary  practices,  difficulties,  or  lacks 

□  Trade  studies 

□  Quality  Function  Deployment  (QFD) 

□  Criteria  weights 

□  Delphi  method 

□  Group  techniques 
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Q2.ll 

Transition  to 
Operations  and 
Support 

Probing  Questions 


CMMI  Practices 


Additional  Topics 


How  does  your  program  provide  for  the  transition  of  the  prod¬ 
uct  to  the  end  user  and  the  eventual  support  organization  and 
accommodate  lifecycle  evolution? 


•  How  does  your  program  prepare  for  transition  to  operations  and 
support? 

•  How  does  your  program  ensure  acquired  products  are  transi¬ 
tioned  to  operations  and  support  based  on  transition  criteria? 

•  Who  is  responsible  for  maintaining  the  system  post  deployment 
and  how  do  they  participate  on  the  program? 

•  Who  is  the  user  representative  and  how  do  they  participate  on 
the  program? 

•  How  will  major  enhancements  be  handled  and  who  is  responsi¬ 
ble  for  providing  the  additional  capability? 

Does  the  program  perform  any  of  the  following  practices?  Does  the 
program  perform  any  particularly  well?  Are  their  some  the  program 
needs  or  wants  to  improve? 

•  Establish  a  Transition  Strategy 

•  Establish  Product  Transition  Plans 

•  Establish  Operations  and  Support  Training  Requirements 

•  Establish  Lifecycle  Resource  Requirements 

•  Identify  Support  Responsibility 

•  Establish  Enhancement  Criteria 

•  Establish  Transition  Criteria 

•  Evaluate  Product  Readiness 

•  Evaluate  Personnel  Readiness 

•  Analyze  Results  and  Take  Action 

Potential  sources  of  exemplary  practices,  difficulties,  or  lacks 

□  Roles  and  responsibilities 

□  Stakeholder  involvement 

□  Transition  strategies  and  plans 

□  Availability  of  training 

□  Access  to  maintenance  tools  and  environments 

□  Access  to  simulators  and  integration  facilities 

□  Capability  of  potential  support  organization 

□  Operational  impacts  due  to  new  system 
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Q2.12 

Organizational 
Environment  for 
Integration 

Probing  Questions 


CMMI  Practices 


Additional  Topics 


How  does  your  organization  provide  an  Integrated  Process  and 
Product  Development  infrastructure  and  manage  people  for 
integration  across  disciplines? 


•  How  does  your  organization  provide  the  infrastructure  to  allow 
multi-disciplinary  teams  to  successfully  function? 

•  How  does  your  organization  the  provide  management  mecha¬ 
nisms  to  allow  multi-disciplinary  teams  to  succeed? 

•  Does  management  provide  special  collaboration  tools  to  allow 
cross-functional  teams  to  work  as  a  virtual  team? 

•  How  are  issues  and  risks  identified  and  addressed  within  the 
teams? 

•  How  does  the  organization  establish  a  shared  vision  across 
teams? 

Does  the  program  perform  any  of  the  following  practices?  Does  the 

program  perform  any  particularly  well?  Are  their  some  the  program 

needs  or  wants  to  improve? 

•  Establish  the  Organization’s  Shared  Vision 

•  Establish  an  Integrated  Work  Environment 

•  Identify  IPPD-Unique  Skill  Requirements 

•  Establish  Leadership  Mechanisms 

•  Establish  Incentives  for  Integration 

•  Establish  Incentives  to  Balance  Team  and  Home  Organization 
Responsibilities 

Potential  sources  of  exemplary  practices,  difficulties,  or  lacks 

□  Collaboration  tools 

□  Clearly  defined  roles  and  responsibilities 

□  Team  leadership 

□  Training  on  working  in  multi-discipline  teams 

□  Encouragement  and  reward  systems  for  teaming 

□  Workload  balancing 
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Q2.13 

Requirements 

Management 

Probing  Questions 


CMMI  Practices 


Additional  Topics 


How  does  your  program  manage  the  requirements  of  the  pro¬ 
gram’s  product,  and  keep  the  program’s  plans  and  work  prod¬ 
ucts  consistent  with  them? 

•  How  are  requirements  managed  so  that  they  accurately  reflect 
the  programs  plans  and  products? 

•  How  does  your  program  ensure  that  requirements  management 
is  institutionalized  as  a  managed  process? 

•  How  does  your  program  maintain  requirements  traceability? 

•  How  does  your  program  analyze  requirements  quality? 

•  How  does  your  program  track  work  effort  against  require¬ 
ments? 

Does  the  program  perform  any  of  the  following  practices?  Does  the 
program  perform  any  particularly  well?  Are  their  some  the  program 
needs  or  wants  to  improve? 

•  Obtain  Understanding  of  Requirements 

•  Obtain  Commitment  to  Requirements 

•  Manage  Requirement  Changes 

•  Maintain  Bi-Directional  Traceability  of  Requirements 

•  Identify  Inconsistencies  between  Project  Work  and  Require¬ 
ments 

Potential  sources  of  exemplary  practices,  difficulties,  or  lacks 

□  Requirements  traceability  matrix 

□  Requirements  management  tool,  database 

□  Requirements  issues  and/or  action  items 

□  Results/minutes  of  requirements  reviews 

□  Updated  plans,  products,  activities 

□  People  assigned  responsibility  and  trained 

□  Requirements  metrics 

□  Evidence  of  adequate  resources  and  tools 

□  Requirements  management  plan 
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Q2.14 

Requirements 

Development 

Probing  Questions 


CMMI  Practices 


Additional  Topics 


How  does  your  program  produce  and  analyze  customer,  prod¬ 
uct,  and  product  component  requirements? 


•  How  does  your  program  collect  and  translate  stakeholder 
needs,  expectations,  constraints,  and  interfaces  into  customer 
requirements? 

•  How  does  your  program  refine  and  elaborate  customer  re¬ 
quirements  to  develop  product  requirements  for  the  product  life 
cycle? 

•  How  does  your  program  ensure  that  the  requirements  are  ana¬ 
lyzed  and  validated  and  that  the  required  functionality  is  devel¬ 
oped? 

•  How  does  your  program  ensure  that  requirements  development 
is  institutionalized  as  a  defined  process? 

•  How  was  this  process  defined  and  where  is  it  documented? 

Does  the  program  perform  any  of  the  following  practices?  Does  the 

program  perform  any  particularly  well?  Are  their  some  the  program 

needs  or  wants  to  improve? 

•  Elicit  Needs 

•  Transform  Stakeholder  Needs,  Expectations,  Constraints,  and 
Interfaces  into  Customer  Requirements 

•  Establish  Product  and  Product  Component  Requirements 

•  Allocate  Product  Component  Requirements 

•  Identify  Interface  Requirements 

•  Establish  Operational  Concepts  and  Scenarios 

•  Establish  a  Definition  of  Required  Functionality 

•  Analyze  Requirements 

•  Evaluate  Product  Cost,  Schedule  and  Risk 

•  Validate  Requirements  with  Comprehensive  Methods 

Potential  sources  of  exemplary  practices,  difficulties,  or  lacks 

□  Functional  architecture 

□  Organizational  process 

□  Technical  Solutions  Plan 

□  Requirements  for  verification  process 

□  Needs  and/or  technical  performance  requirements 

□  New/derived  and/or  product  requirements 

□  Evidence  of  adequate  resources  and  tools 

□  Customer  requirements  documents  and/or  baseline 
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Q2.15 

Verification 


How  does  your  program  assure  that  selected  work  products  meet 
their  specified  requirements? 


Probing  Questions 


CMMI  Practices 


Additional  Topics 


•  How  does  your  program  establish  and  maintain  strategies,  re¬ 
quirements,  plans,  and  environment  for  verification? 

•  How  does  your  program  perform  peer  reviews  on  key  work 
products? 

•  How  does  your  program  ensure  that  work  products  are  verified 
against  their  specified  requirements? 

•  How  does  your  program  ensure  that  verification  is  institutional¬ 
ized  as  a  defined  process. 

•  How  does  your  program  ensure  that  the  Product  Verification 
process  is  being  followed,  overall  and  on  a  day  to  day  basis? 

•  How  is  corrective  action  taken  when  discrepancies  are  found  in 
work  products?  When  is  re-verification  performed? 

•  Do  you  conduct  formal  inspections  on  key  work  products?  Are 
other  peer  reviews  performed? 

Does  the  program  perform  any  of  the  following  practices?  Does  the 
program  perform  any  particularly  well?  Are  their  some  the  program 
needs  or  wants  to  improve? 

•  Establish  a  Verification  Strategy 

•  Establish  the  Verification  Environment 

•  Establish  Detailed  Verification  Plans 

•  Prepare  for  Peer  Reviews 

•  Conduct  Peer  Reviews 

•  Analyze  Peer  Review  Data 

•  Perform  Verification 

•  Analyze  Verification  Results  and  Identify  Corrective  Actions 

•  Perform  Re-Verification 

Potential  sources  of  exemplary  practices,  difficulties,  or  lacks 

□  Project  Verification  Plan 

□  Verification  traceability  matrix 

□  Requirements  traceability  matrix 

□  Effort  expended  for  verification  and  rework 

□  Work  product  inspection  logs/results 

□  Verification  logs 

□  Organizational  policy  for  product  verification 

□  Performance  and  verification  analysis  reports 

□  DR  problem  report  trends/aging,  etc. 
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Q2.16 

Validation 


How  does  your  program  confirm  that  a  product  fulfills  its  in¬ 
tended  use  when  placed  in  its  intended  environment? 


Probing  Questions 


CMMI  Practices 


Additional  Topics 


•  How  does  your  program  establish  and  maintain  requirements, 
strategies,  and  preparation  activities  for  validation? 

•  How  does  your  program  validate  product  requirements  and 
products  against  their  operational  needs? 

•  How  does  your  program  ensure  that  validation  is  institutional¬ 
ized  as  a  defined  process. 

•  What  work  products,  end  products,  operational  elements,  and 
other  life  cycle  elements  does  your  program  validate? 

•  How  does  your  program  ensure  that  the  Validation  process  is 
being  followed? 

Does  the  program  perform  any  of  the  following  practices?  Does  the 
program  perform  any  particularly  well?  Are  their  some  the  program 
needs  or  wants  to  improve? 

•  Establish  a  Validation  Strategy 

•  Establish  the  Validation  Environment  (CL  2) 

•  Define  Detailed  Validation  Procedures  (CL  3) 

•  Perform  Validation 

•  Capture  and  Analyze  Validation  Results 

Potential  sources  of  exemplary  practices,  difficulties,  or  lacks 

□  Project  Validation  Plan 

□  Validation  strategy 

□  Effort  expended  for  validation 

□  Validation  reports 

□  Detailed  description  of  validation  environment 

□  Inspection  and  simulation  analysis  results 

□  Number  of  validation  tests  versus  planned 

□  Acceptance  test  cases,  procedures,  and  results 
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Process  Institutionalization 

Examine  process  institutionalization  for  each  process  area: 

•  Establish  and  Maintain  an  Organizational  Policy 

•  Plan  the  Process 

•  Provide  Resources 

•  Assign  Responsibility 

•  Train  People 

•  Manage  Configurations 

•  Identify  and  Involve  Relevant  Stakeholders 

•  Monitor  and  Control  the  Process 

•  Objectively  Evaluate  Adherence 

•  Review  Status  with  Higher  Level  Management 

•  Establish  a  Defined  Process 

•  Collect  Improvement  Information 

Relationships 

Q3.  Considering  the  successes  and  issues  talked  about,  how  are 

Relationships  overall  strengths  or  weaknesses  influenced  by: 

•  program-to-program  relationships 

•  program-to-developer  relationships 

•  defined  processes 

•  root  causes  that  are  outside  the  control  of  your  program  and 
more  a  result  of  constraints  imposed  on  the  system  by  such 
things  as,  for  example,  congressional  mandates 

•  other  (if  the  above  categories  are  perceived  by  the  interviewee 
as  not  being  exhaustive) 


Q2.17 

Process 

Institutionalization 
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